Image retrieving method and apparatuses therefor

ABSTRACT

By sequentially inputting images for each frame, sequentially extracting features from the inputted frame images, converting the features sequentially extracted into a feature series corresponding to the inputted frame image series, compressing the feature series in the direction of the time axis, storing the compressed feature series in the storage, sequentially extracting features separately from the images to be retrieved for each inputted frame, sequentially comparing the features of the images to be retrieved for each frame with the stored compressed feature series, storing the progress state of this comparison, updating the stored progress state of the comparison on the basis of a comparison result with the frame features of the succeeding images to be retrieved, and retrieving image scenes matching with the updated progress state from the images to be retrieved on the basis of the comparison result between the updated progress state and the features of the images to be retrieved for each frame, the present invention can retrieve video images on the air or video images in the data base at high speed and enables self organization of video to be classified and arranged on the basis of the identity of partial images of video.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to a retrieving method andapparatuses therefor for video images on the air or video images in adata base or others and more particularly to a video image retrievingmethod and apparatuses therefor for performing high-speed retrieval bythe help of features of video images.

[0003] 2. Description of the Prior Art

[0004] Recently, multi-media information processing systems can storeand indicate various types of information such as video and text tousers. However, with respect to retrieval of them, a retrieving methodusing a language such as a keyword is mainly used. In this case, akeyword assigning operation is necessary and it is extremely expensiveto assign a keyword to each frame of video having a large amount ofinformation. Furthermore, since a keyword is freely assigned by a database constructor, there is a problem imposed that when the viewpoint ofa user is different from that of the data base constructor, the keywordwill be useless. In these circumstances, a request for retrieval from aunique image feature in addition to the keyword is made. However, toretrieve information on the basis of the feature of an image, ahigh-speed comparison art between the video feature comprising enormousframes and the feature for the queried image is necessary. As ahigh-speed comparison art only applicable to video images, “Videoretrieving method and apparatuses therefor” is proposed in JapanesePatent Application Laid-Open 7-114567. This method does not compare allthe frames but compares only an image at the time of changing of cut ofimages so as to reduce the processing amount. By doing this, the highspeed also suited to comparison of images on the air is realized. On theother hand, there is a problem imposed that a scene comprising only onecut or a scene in which the cut change timing varies with editing beforeor after cannot be compared satisfactorily. Furthermore, duringretrieval, scenes other than the scene specified as a retrieval key arenot searched in the same way as with other general data base systems, sothat whenever scene retrieval becomes necessary, it is necessary torepeatedly compare a very large amount of video information from thebeginning thereof to the last. The scene comparison process includes anumber of processes such as processes to be performed commonly even ifthe scene to be retrieved is different as well as the feature extractionand reading processes and repetitive execution of such a process is ofno use.

SUMMARY OF THE INVENTION

[0005] An object of the present invention is to solve the aforementionedproblems and to provide an image retrieving method for comparing thefeature of a target image to be retrieved and the feature of a sampleimage to be prepared for query at high speed without performing akeyword assigning operation for image retrieval and for detecting thesame segment with the frame accuracy. A target image on the air or inthe data base is applicable.

[0006] Another object of the present invention is to provide a methodfor detecting the same scene existing in the target image regardless ofwhether it is specified as a retrieval key beforehand in the same way atthe same time with input of the target image.

[0007] Still another object of the present invention is to provide avideo camera for comparing, when recording an image series inputted frommoment to moment during picking up of images, those images with recordedimages and recording them in association with matched images.

[0008] To accomplish the above objects, the present invention is asignal series retrieving method and apparatuses therefor in aninformation processing system comprising a time sequential signal inputmeans, a time sequential signal process controller, and a storage,wherein the method and apparatuses sequentially input time sequentialsignals, sequentially extract features in each predetermined period ofthe inputted time sequential signals, convert the features sequentiallyextracted into a feature series corresponding to the inputtedpredetermined period series, compress the feature series in thedirection of the time axis, store the compressed feature series in thestorage, sequentially extract features from the time sequential signalsto be retrieved in each predetermined period of the inputted timesequential signals, sequentially compare the features of the timesequential signals to be retrieved in each predetermined period with thestored compressed feature series, store the progress state of thecomparison, and retrieve a signal series matching with the progressstate from the time sequential signals to be retrieved on the basis ofthe comparison result between the stored progress state of thecomparison and the features of the time sequential signals to beretrieved in each predetermined period.

[0009] More concretely, the present invention divides a video image tobe compared into the segment-wise so that the feature of each frame isset in the variation width within the specific range respectively,extracts one or a plurality of features in each segment, stores it orthem in correspondence with the address information indicating theposition in the image in the segment, then sequentially inputs frameimages one by one from video images to be retrieved, and when thefeature series at an optional point of time in which the features of theframe images are sequentially arranged and the feature series in whichthe features in the segments constituting the stored images aresequentially arranged in each segment length have portions equal to ormore than the specific length which can be decided to be mutuallyequivalent to each other, detects the portions as a same image. In thiscase, when they are equivalent to each other from the top of a segment,the present invention obtains the address information corresponding tothe segment and when they are decided to be equivalent to each otherfrom halfway of a segment, the present invention obtains the relativeposition from the top of the segment, and outputs a corrected value ofthe address information corresponding to the segment as a retrievalresult. Furthermore, the present invention collects a frame image seriesinputted as a retrieval target in each segment so that the features ofthe frames are set in the variation width within the specific range,extracts one or a plurality of features in each segment, also stores theinformation corresponding to the address information indicating theposition in the target image in the segment, and adds it to the targetimages to be compared next. Furthermore, with respect to the inputtedfeature series, when there are a plurality of video portions which aredetected to be the same, the present invention groups them, associatesthem to each other, and stores them.

[0010] An apparatus realizing the aforementioned retrieving methodcomprises a means for dividing an optional image into the segment-wiseso that the feature of each frame is set in the variation width withinthe specific range respectively, a means for extracting one or aplurality of features in each segment, a means for storing it or them incorrespondence with the address information indicating the position inthe image in the segment, a means for sequentially inputting frameimages one by one from images to be retrieved, a means for retaining thefeature series at an optional point of time in which the features of theframe images are sequentially arranged, a means for generating thefeature series in which the features in the segments constituting thestored images are sequentially arranged in each segment length, and ameans for deciding whether the feature series have portions equal to ormore than the specific length which can be decided to be mutuallyequivalent to each other. The present invention also has a means forobtaining, when they are decided to be equivalent to each other from thetop of a segment, the address information corresponding to the segment,when they are decided to be equivalent to each other from halfway of asegment, obtaining the relative position from the top or the segment,and outputting a corrected value of the address informationcorresponding to the segment as a retrieval result. Furthermore, thepresent invention has a means for collecting a frame image seriesinputted as a retrieval target in each segment so that the features ofthe frames are set in the variation width within the specific range, ameans for extracting one or a plurality of features in each segment, anda means for also storing the information corresponding to the addressinformation indicating the position in the target image in the segmentand adding it to the target images to be compared next. Furthermore,with respect to the inputted feature series, when there are a pluralityof scenes which are detected to be the same, the present invention has ameans for grouping them, associating them to each other, and storingthem.

[0011] The foregoing and other objects, advantages, manner of operationand novel features of the present invention will be understood from thefollowing detailed description when read in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012]FIG. 1 is a block diagram of a system for executing an embodimentof the present invention.

[0013]FIG. 2 is a block diagram of a process for executing an embodimentof the present invention.

[0014]FIG. 3 is a schematic view showing the feature extracting methodof an embodiment of the present invention.

[0015]FIG. 4 is a schematic view showing the feature comparing method ofan embodiment of the present invention.

[0016]FIG. 5 is a drawing showing an example of feature comparison flowof an embodiment of the present invention.

[0017]FIG. 6 is a schematic view showing an example of the conventionalcomparing method.

[0018]FIG. 7 is a schematic view for explaining the comparing method ofan embodiment of the present invention.

[0019]FIG. 8 is a schematic view for explaining the comparing method ofan embodiment of the present invention.

[0020]FIG. 9 is a block diagram of a process for executing an embodimentof the present invention.

[0021]FIGS. 10A and 10B are flow charts of an embodiment of the presentinvention.

[0022]FIG. 11 is a drawing showing the feature table structure used inan embodiment of the present invention.

[0023]FIG. 12 is a drawing showing the candidate list structure used inan embodiment of the present invention.

[0024]FIG. 13 is a drawing showing the candidate structure used in anembodiment of the present invention.

[0025]FIG. 14 is a drawing showing the retrieval result table andretrieval segment structure used in an embodiment of the presentinvention.

[0026]FIG. 15 is a schematic view of a video recorder system applying anembodiment of the present invention.

[0027]FIG. 16 is a drawing showing a display screen example during imageretrieval of self organization of video by the present invention.

[0028]FIG. 17 is a drawing showing a display screen example during imageretrieval of self organization of video by the present invention.

[0029]FIG. 18 is a drawing showing a display screen example during imageretrieval of self organization of video by the present invention.

[0030]FIG. 19 is a schematic block diagram when the present invention isapplied to a video camera.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] An embodiment of the present invention will be explainedhereunder by referring to the drawings.

[0032]FIG. 1 is an example of a schematic block diagram of the systemconfiguration for realizing the present invention.

[0033] Numeral 1 indicates a display such as a CRT, which displays anoutput screen of a computer 2. When the output of the computer is voice,the computer 2 outputs it via a speaker 13. An instruction to thecomputer 2 can be issued using a pointing device 3 and a keyboard 4. Avideo reproducing apparatus 5 is an optical disk or a video deck. Avideo signal outputted from the video reproducing apparatus 5 issequentially converted to digital image data by a video input device 6and sent to the computer. In certain circumstances, an image on the aircan be fetched and a video signal from a broadcast receiver 7 isinputted to the video input device 6. When a video server recording animage as digital data or digital video is used instead of the videoreproducing apparatus 5, the video input device 6 is unnecessary or afunction for expanding compressed and recorded image data and convertingit to incompressed image data is controlled. If the broadcast is of adigital system, the same may be said with the broadcast receiver 7.Inside the computer, digital image data is inputted to a memory 9 via aninterface 8 and processed by a CPU 10 according to a program stored inthe memory 9. When video handled by the CPU 10 is sent from the videoreproducing apparatus 5, a number (frame No.) is sequentially assignedto each frame image starting from the top of video. When a frame numberis sent to the video reproducing apparatus by a control line 11, theapparatus can control so as to reproduce the video of the scene. Whenvideo is sent from the broadcast receiver 7, no frame number isassigned, so that the apparatus records a sequence number or timestarting from a process start time of 0 as required and uses it insteadof the frame number. Various informations can be stored in an externalinformation storage 12 as required by the internal process of thecomputer. Various data created by the process which will be explainedhereunder is stored in the memory 9 and referred to as required.

[0034]FIG. 2 is a whole block diagram showing the process outline of theimage retrieval process of the present invention. This process isexecuted inside the computer 2. The process program is stored in thememory 9 and executed by the CPU 10. Hereunder, the process will beexplained on the assumption that each unit is described as a softwareprocedure to be executed by the CPU 10. However, needless to say, afunction equivalent to this procedure can be realized by hardware. Inthe following explanation, the processes performed by the software areblocked for convenience. Therefore, for example, in FIG. 2, the inputunit for queried image indicates an input process for queried image. Inthis embodiment, an image of the scene to be found out (hereinafter,called a queried image) 100 is sequentially inputted for each frame byan input unit for queried image 102 beforehand prior to retrieval andtemporarily stored in the memory 9. A frame feature extractor 106extracts a feature 8 from a frame image 104 in the memory 9. A featuretable generator 110 pairs up the feature and the top frame number foreach segment of a string of features when the feature is within theallowable variation range, creates a feature table 112, and records itin a storage 114. Also an image 116 to be retrieved is sequentiallyinputted for each frame by an input unit for target image to be compared118 in the same way as with a queried image and temporarily stored inthe memory 9. A frame feature extractor 122 extracts a feature 124 froma frame image 120 in the memory 9. In this case, the frame featureextractor 122 performs the exactly same process as that of the framefeature extractor 106. A feature comparator 130 compares the newest timesequential array of the features 124 sequentially sent from the framefeature extractor 122 with a stored feature table 300 (the data contentis the same as that of the feature table 112) for consistency. Theprogress state of the comparison is stored in the storage 126 in theform of a candidates list 400 which will be described later and updatedevery input of a new frame. If the features are consistent with eachother, the image segment corresponding to the feature table is outputtedto a storage 128 or the other processor as a retrieved result table 600which will be described later. If any name and attribute are associatedwith the retrieved image in this case, it is naturally possible tooutput the name and attribute.

[0035] Next, the process performed by each unit mentioned above will beexplained more in detail.

[0036]FIG. 3 shows a series of flow (100 to 114) from input of a queriedimage to creation of a feature table. The object of this process is tocompress queried images to a minimum quantity of information which canrepresent the features thereof so as to store more types of queriedimages and compare them in real time at one time. Concretely, featuresare extracted from frame images sequentially inputted first. In thiscase, the feature is explained as information which can be representedby several bytes such as the mean color of the whole frame images. As afeature, in addition to it, patterns generally known such as the shapeof the boundary line and texture of a specific image can be widelyapplied. Furthermore, the time sequential array of obtained features iscollected for each segment within the allowable variation range and onefeature is represented in each segment. A′ or A″ shown in the drawingindicates that assuming A as a standard, the absolute value of thedifference of the feature value of A′ or A″ from that of A is less thana specific threshold value. To each frame of inputted images, framenumbers are sequentially assigned such as t₁, t₂, t₃, - - - , and theframe numbers t_(i), t_(j), t_(k), - - - of the top frame of eachsegment and the features A, B, C, - - - are paired up, and a list isgenerated as a feature table. In this case, video comprises 30 frameimages per second, so that although depending on the kind of an image tobe searched for, assuming that the mean segment length is 10 frames, apermutation pattern comprising 10 or more features can be obtained evenfrom a scene in only several seconds. Furthermore, if the length of eachsegment is added to the restrictions, the number of permutations andcombinations of feature tables becomes extremely large in this case anda performance for sufficiently specifying one scene even in many imagescan be expected.

[0037]FIG. 4 schematically shows the situation of comparison (thefeature comparison process 130) between the video image to be retrievedand the queried image stored beforehand. As mentioned above, withrespect to target images to be retrieved, frame image are sequentiallyinputted and features are extracted (116 to 124). On the other hand,with the queried images compressed in the form of feature table, thefeatures are arranged in the length of each segment and the featureseries is returned from the run-wise to the frame-wide during comparison(130). In the case of comparison, a queried image having a featureseries matching with the feature series in a length more than thespecific threshold value which has the newest frame just inputted fromthe target image as a last end is returned as a retrieved result. Inthis case, not only a complete match but also a partial match of thefeature series are detected and when the length of the matched part ismore than the threshold value, it is also returned as a retrievedresult. By doing this, also a scene in which the length is delicatelydifferent due to editing can be correctly retrieved.

[0038]FIG. 5 shows the comparison process of the present invention morein detail. If, when a feature series in an indefinite length asmentioned above is compared, the comparison is simply executed, it isnecessary to repeat a comparison on the assumption of various framelengths as shown in FIG. 6 whenever a frame image is newly inputted fromthe target image. The number of inter-frame comparisons in this case isextremely enormous as shown in the drawing and the comparison process isnot suited especially to comparison in real time such that new framesare inputted one after another at a rate of once per {fraction (1/30)}seconds. The reason is that the comparison process is executed quiteindependently of the previous comparison process every input of a frameand even if a match of a certain degree of length is ascertained by thejust prior process, the information cannot be applied to the nextcomparison process. Therefore, the present invention takes an approachto reduce the comparison process to be performed for one frame input andto stepwise perform the comparison process so as to supplement theprevious process every frame input. Concretely, the comparison isexecuted as indicated below.

[0039] (1) When a frame is inputted from the target image, it issearched whether there is a feature which is the same as that of theframe in the queried image and all found frames are temporarily storedas candidates.

[0040] (2) When the next frame is inputted from the target image, it ischecked whether the feature of the frame matches with the feature of theframe immediately after the frame stored as a candidate immediatelybefore.

[0041] (3) When they match with each other, the frame is set as acandidate together with the frame stored as a candidate immediatelybefore and when they do not match with each other, the frame is excludedfrom a candidate and a frame having the same feature as that of the justinputted frame is newly added as a candidate. In this case, if the frameexcluded from a candidate is kept consistent for the length (the numberof frames) more than the specific threshold value till that time, thematched segment with the frame set at the top is outputted as aretrieved result.

[0042] (4) The aforementioned operations are repeated.

[0043] The comparison principle of the present invention will beconcretely explained hereunder by referring to the example shown in FIG.5.

[0044] Firstly, a new frame is inputted from the target image and theframe (1) in which the feature X is obtained will be considered. Sincethere is not the feature X in the queried image, nothing is performed.The same may be said with the frame (2). When the frame (3) is inputtedand the feature A′ is obtained, there is the feature A matching with A′in the queried image, so that all the frames {circle over (1)} to{circle over (4)} having the feature A in the queried image are set ascandidates. Depending on the appearing condition of features of framesto be inputted hereafter from the target image, any of these candidateframes has a possibility that one segment with the frame set at the topbecomes a scene to be retrieved. In the lower table shown in FIG. 5,{circle over (1)} to {circle over (4)} written on the line of Frame (3)indicate frames in the queried image which are selected as candidates atthis point of time. Also in the next frame (4), the feature A′ isobtained. Firstly, all the frames selected as candidates at thepreceding step are checked whether the next frames match in feature. Asa result, the frames {circle over (1)} to {circle over (3)} match infeature but the frame {circle over (4)} does not match in featurebecause the feature of the next frame {circle over (4)} is changed to B.The portion of x marked on the fourth line in the table indicates it andthe frame {circle over (4)} selected as a candidate in the frame (3) isexcluded from a candidate at this point of time. At the same time, ascandidates in the frame (4), {circle over (1)} to {circle over (4)}which are the same as those of (3) are newly added on the fourth line inthe table. Although the frames {circle over (1)} to {circle over (4)}added on the line (3) are the same as the frames {circle over (1)} to{circle over (4)} added on the line (4), they are handled as differentcandidates as comparison candidates. Furthermore, B is obtained in theframe (5) and {circle over (1)} and {circle over (2)} selected ascandidates in (3) and {circle over (1)} to {circle over (3)} selected ascandidates in (4) are excluded from candidates. In the same way, {circleover (5)} and {circle over (6)} are selected as candidates at this pointof time. When the aforementioned process is repeated whenever a frame isinputted from the target image, candidates matching continuously up tothe step of the frame (8) are only {circle over (3)} selected as acandidate in (3), {circle over (4)} selected as a candidate in (4),{circle over (5)} selected as a candidate in (5), {circle over (6)}selected as a candidate in (6), and {circle over (7)} selected as acandidate in (7). At the point of time that the frame (9) is inputtedand no comparison can be made, it is found that the frames (3) to (8) ofthe target image and the queried images {circle over (3)} to {circleover (8)} have a longest matching segment. These results match with thecomparison results when the comparison of scenes is checked bysequentially changing the length with the frame (8) as starting pointusing the conventional method previously shown in FIG. 6. In the case ofFIG. 6, assuming the number of frames of queried images as n, therepetition time of comparison between the frames to be executed everyone frame input is n(n+1)(n+2)/6 times as shown in FIG. 6 and the orderof the calculated value is 0(n³). However, according to this method,only the sum of (1) the repetition time c of checking for a match of thefeature of a newly inputted frame with the feature of the next frame tothe candidate frame and (2) the repetition time n of checking whetherthere is the same feature as that of the newly inputted frame in thequeried images is acceptable, and generally n>>c, and the order is 0(n).This difference is cased by use of the inductive method for obtainingthe result of adding the current frame on the basis of the processingresult up to the just prior frame. n can be made smaller than theoriginal number of frames by use of the aforementioned feature table anda quicker comparison can be expected. Furthermore, the retrieved resultcan be clearly positioned with the frame accuracy.

[0045] In the above explanation, a case of one queried image is assumed.However, the principle can be also applied to a plurality of queriedimages without trouble. For comparison every frame input, it isdesirable only to repeat the aforementioned process for the number ofqueried images. However, as shown in FIG. 7, although the same imagepart is included in each of the queried images, they may be delicatelydifferent in the longitudinal direction due to an effect of a differentediting way. In the drawing, three kinds of ways {circle over (1)},{circle over (2)}, and {circle over (3)} are shown. The same may be saidwith a case that a plurality of same image parts are included in onequeried image. When only whether there is a matched part in the queriedimage is necessary, no problem is imposed. However, depending on theobject of retrieval, also the classification may be required on thebasis of the accurate position and length of the matched segment. Inthis case, it is necessary to clearly output what segment matches withwhat segment as a retrieved result. When there is an overlapped part asshown in No. 2 and No. 3 in the drawing, it is necessary to indicate theoverlapped part in consideration of the inclusion relationship. Themethod of the present invention can process also this problem at highspeed without changing the basic comparison principle. In the comparisonprocess of this method, it is described that when a frame is inputtedfrom the target image and the feature thereof is obtained, a group offrames having the same feature as that of the target image is selectedas candidates from the queried images. In this case a group of matchedsegments with the frames selected as candidates at the same time set atthe top which reach a length more than the detected threshold value isimages which are equal to each other. In the example shown in FIG. 7,the segment {circle over (2)} exists in each of the three queried imagesand all the top frames of the segments of the queried images areselected as candidates at the same time when the frame corresponding tothe top of the segment {circle over (2)} is inputted from the targetimage. Although there is the possibility that there are other frames tobe selected as candidates at the same time, they are excluded fromcandidates before they reach a length more than the detected thresholdvalue. They reach the end of the segment {circle over (2)} and when thenext frame is compared, the matched segment in the queried images of No.1 and No. 3 is excluded from a candidate. The target image stillcontinues the match with No. 2. However, the segment {circle over (2)}is decided for the present and it is outputted as a retrieved resultthat {circle over (2)} is detected in the queried images No. 1 to No. 3.However, even if the segment {circle over (2)} ends, the queried imageNo. 2 continuously remains as a candidate because also the next frame isstill matched with the target image and finally the segment {circle over(3)} is decided. Even if there is a segment on this side of {circle over(2)} like {circle over (1)} the matched segment is detected and decidedin the same way. As mentioned above, according to the method of thepresent invention, only by performing a brief check when a segment isselected as a candidate or excluded from a candidate, scenes of variousvariations delicately different in the longitudinal direction can bediscriminated and detected respectively with the comparison processingamount every frame input kept small.

[0046] In the above explanation, a case that queried images are preparedbeforehand and then the target image is retrieved is used. However, thismethod can be applied even if the queried images are just target images.FIG. 8 shows a conceptual diagram thereof. Target images are inputted,and all of them are stored, and they are handled as if they are theaforementioned queried images. It can be realized by the block diagramshown in FIG. 9. Although it is almost similar to the block diagramshown in FIG. 2, the queried images are the same as the target images,so that the process up to extraction of frame features can be shared andthe frame feature 108 is distributed for storage and comparison. By thismechanism, the part of target images inputted past where the newestimage part {circle over (1)} inputted from the target images appears canbe detected at the same time with input. If scenes appear several timespast, all of them are detected at the same time on the aforementionedcomparison principle, so that they are collected, classified, andarranged for each detected same scene. So to speak, self organization ofvideo is automatically realized in real time. For example, if thepresent invention is applied to an apparatus for recording TV programsfor several weeks to which a memory capacity for storing all TV programsfor several weeks is installed, the same image is generally outputtedevery time at the opening of a program, so that by detecting the imageand collecting the images before and after it, the programs can bearranged in real time at the same time with recording. If it is foundthat there are a plurality of same scenes, it is possible to leave onlyone image and erase the residual images by leaving only pointers, sothat the use efficiency of media for recording can be improved. Althoughalso a commercial message is one of images outputted repeatedly, to playback a recorded program, the commercial message can be automaticallyskipped as required. In this case, by use of the commercialcharacteristic that the length is just 15 seconds or 30 seconds, thedecision performance as to whether it is a commercial message isimproved.

[0047] In the above explanation, the process of realizing the blockdiagram shown in FIG. 9 can be represented more concretely by the flowcharts shown in FIGS. 10A and 10B. Also the process of realizing theblock diagram shown in FIG. 2 is self-evident from FIGS. 10A and 10B. Inthe above explanation, for simplicity, the feature of the queried imageis returned from the run-wise to the frame-wise once and then compared.However, to make the specification closer to the practical use, a methodof comparison in the run-wise state will be indicated hereunder.

[0048] Firstly, at Step 200, the apparatus and various variables areinitialized. The variables mc and mm are set to 0. Next, a frame imageis inputted from the target image (Step 202) and the feature F isextracted from the frame image (Step 204). The feature F uses the meanof colors of all pixels existing in the frame image. The color of eachpixel is represented by the three components R, G, and B, and withrespect to the value of each component, the values on the whole screenare averaged respectively, and a set of three values (Ra, Ga, Ba) isobtained, and this set is assumed as the feature F. If a first frame isinputted, a feature table structure 300 shown in FIG. 11 is newlygenerated and F is written into 302 as a feature of the first segment(segment No. 1). In this case, the frame number is also written into 304as a pair. The feature table generated like this will function hereafterfor the already mentioned queried image. In this case, the variable mcindicating the maximum value of the segments stored in the feature tablestructure 300 is incremented by one and the program is returned to Step202 as it is. On the other hand, if the second frame or a subsequentframe is inputted, Step 206 is executed. At Step 206, the feature FC ofthe newest segment (the segment of the segment number mc−1) stored inthe feature table and the current feature F are compared and it isdecided whether the difference is smaller than the threshold value CTH.In this case, although the feature is a set of three values as mentionedabove, only when the differences between the three values are allsmaller than the threshold value CTH, it is represented that thedifference is smaller than the threshold value CTH. If the difference issmaller than the threshold value CTH, it is decided that the framecurrently inputted can be collected in the same segment as that of thejust prior frames and the program goes to Step 208. At Step 208, theloop counter i is reset to 0. i is incremented by 1 every time at Step226 and Steps 210 to 224 are repeated until i becomes larger than mm. Inthis case, mm indicates the number of candidates at the stage ofcontinuous inspection among all images (stored as the feature table 300)inputted until now on the assumption that there is the possibility thatthe part is the same as an image being newly inputted at present. Astructure 500 for storing the status variable indicating the inspectionstage of each of all candidates is generated and managed by a candidatelist structure 400 as shown in FIG. 12. Pointers to the candidatestructure 500 are stored in the candidate list structure 400 anddynamically added or deleted during execution. FIG. 13 shows theconstitution of the candidate structure 500 and the segment number whenit is registered as a candidate is stored as a starting segment numberof comparison 502 and the segment number which starts from the segmentand is a target of comparison at present is stored as a target segmentnumber of comparison 504. A matching frame number counter 506 indicatesthe repetition time of matching since selected as a candidate, that is,the matching segment length. A starting frame offset for comparison 508is a variable necessary for positioning with the frame accuracy byperforming comparison in run-wise, which will be described later.Pointers to starting candidates of simultaneous comparison 510 connect agroup of candidates simultaneously registered to each other in theconnection list format and candidates simultaneously registered can besequentially traced by referring to 510. At Step 210, the program checkswhether the comparison of the candidate i (indicated as a means of thei-th candidate among the mm candidates) is completed to the end of thesegment which is a comparison target at present. When the frame numberobtained by adding the matching frame number counter 506 to the framenumber of the segment indicated by the starting segment number ofcomparison 502 reaches the frame number of the segment next to thesegment which is a comparison target at present, it is found that thecomparison reaches the end. If it does not, the program increments thematching frame number counter of the candidate i by one (Step 216) andgoes to Step 226. If it does, the program refers to the feature of thesegment following the segment which is a comparison target at presentand checks whether the difference between the feature and F is smallerthan the threshold value STH (Step 212). If the difference is smallerthan the threshold value STH, the program changes the segment to becompared to the next segment and continues the comparison (Step 214). Bydoing this, even if the segment changing location is different from theinput image, it can be stably compared. This is a necessary processbecause, since a video signal may be changed due to noise during imageinput and characteristics of the apparatus, the changing point of thesegment is not always the same even if the same image is inputted. Thereason for use of the threshold value STH which is different from thethreshold value CTH deciding the segment change timing is that thechange of an image is absorbed in the same way and a stable comparisonis executed. On the other hand, at Step 212, when the difference islarger than the threshold value STH, the program checks whether thedifference between the feature of the segment which is a comparisontarget at present and the current feature F is smaller than thethreshold value STH (Step 218). If the difference is smaller than thethreshold value STH, the program goes to Step 226 without doinganything. The reason is that since a segment is selected as a candidatenot in frame-wise but in segment-wise and the features do not alwaysmatch with each other starting from the top of the segment, while aninput image having the same feature as that of the segment which is acomparison target at present is obtained, the program only waits bypositioning for the present. If the difference is larger than thethreshold value STH, it is regarded that the features do not match witheach other any more. If the value of the matching frame number counterof the candidate i is larger than the threshold value FTH in this case(Step 220), the program outputs the candidate i as a retrieved scene(Step 222). The program deletes the candidate i from the candidate list(Step 224) and goes to Step 226.

[0049] At Step 206, if the difference is larger than the threshold valueCTH, it is decided that the currently inputted frame cannot be collectedin the same segment as that of the previous frames and a new segment isadded to the feature table 300 (Step 228). In this case, mc isincremented by one and F is substituted for FC. At Step 230, the loopcounter i is reset to 0. i is incremented by one every time at Step 248and Steps 232 to 246 are repeated until i becomes larger than mm. AtStep 232, the program checks whether the comparison of the candidate iis completed to the end of the segment which is a comparison target atpresent. This can be obtained by the same method as that of Step 210. Ifthe comparison reaches the end, the program changes the segment to becompared to the next segment (Step 234) and if it does not, the programdoes nothing. Next, the program checks whether the difference betweenthe feature of the segment which is a comparison target at present andthe newest feature F is smaller than the threshold value STH (Step 236).If the difference is smaller than the threshold value STH, the programincrements the matching frame number counter of the candidate i by one(Step 238) and goes to Step 248. If the difference is larger than thethreshold value STH, the program checks not only one segment immediatelyafter the segment which is a comparison target at present but also thefollowing segments sequentially and checks whether there is a segmenthaving the same feature as the current feature F (Step 240). If thereis, the program changes the next segment to a segment to be compared,substitutes the difference between the frame number of the segment andthe frame number which is attempted to compare at first for the startingframe offset for comparison 508, and goes to Step 248. Also the framenumbers do not always match with each other starting from the top of thesegment, so that the positioning with the frame accuracy can be executedby use of this offset. In this case, if the size of the offset is largerthan the segment length when it is selected as a candidate, the programgoes to Step 242 by the same handling as that when no matching followingsegment is found. If it is not, it is equivalent to the comparisonstarted from a segment behind the segment selected as a candidate firstand in this case, it is expected that in the comparison started from therear segment, a match is smoothly continued and the processing isduplicated. If, when no matching following segment is found, the valueof the matching frame number counter of the candidate i is larger thanthe threshold value FTH (Step 242), the program outputs the candidate ias a retrieved scene (Step 244). The program deletes the candidate ifrom the candidate list (Step 246) and goes to Step 248. When theprocess for all the candidates ends, the program searches all segmentshaving the same feature as that of the currently inputted frame imagefrom the segments stored in the feature table, generates a candidatestructure having these segments as comparison starting segments, andadds it to the candidate list (Steps 250 to 236).

[0050] At Steps 222 and 244 among the aforementioned steps, the programnot only outputs the information of a found scene as it is but also canoutput it in the formats shown in FIG. 14. The retrieved result table600 collects and groups found scenes for each same scene and manages theentry of each group. A group of same scenes is obtained as previouslyexplained in FIG. 7. Each of found scenes is represented by a retrievedsegment structure 700 and the same scenes represent one group in theconnection list format that the scenes have mutually pointers. Pointersto same scenes forming a connection list are stored in 704 and the topframe number of each segment is stored in 702. A pointer to theretrieval segment structure which is the top of the connection listrepresenting a group is stored in 602 as an entry of the group. In thesame group, the segment lengths of all scenes in the group are the same,so that they are paired up with the entry and stored in 604.

[0051] When the aforementioned processes are repeated, a scene whichappeared once in the past is detected the moment it appears once againand the top and length of the segment are positioned with the frameaccuracy. The top of the segment is a frame in which the starting frameoffset for comparison of the candidate structure is added to the framenumber of the segment indicated by the starting segment number ofcomparison of the candidate structure and the length is the value of thematching frame number counter itself. Hereafter, by collecting each samesegment, automatic self organization can be realized. However, in thecase of a scene that a still image continues for a long time, a problemalso arises that by this method reducing the feature of each frame, thecharacteristic time change of the feature cannot be obtained and theprobability of matching with another still image scene by mistakeincreases. If this occurs, needless to say, it can be solved byincreasing the feature for each frame image. Also in the case of a scenethat the feature changes little, even if a shift of several framesoccurs, the features can match with each other. In such a case, aplurality of segments are overlapped and detected in the same range. Asa typical example of it, there is a case that an image just inputtedmatches with a segment a little before in the same cut (one of the unitsconstituting an image, a collected-image segment continuouslyphotographed by a camera). The reason is that the frames in the same cutare well similar to each other on an image basis due to the redundancyof images. If this occurs, by introducing the known detection method forthe cut change timing and performing a process of not regarding as amatch in the same cut, the problem can be avoided.

[0052]FIG. 15 is a conceptual diagram showing an embodiment of a nextgeneration video recorder system using the present invention,particularly the method shown in FIG. 8. The system records video of aTV program and also executes the function of the present invention atthe same time. Address information such as a frame number is assigned toeach frame of video to be recorded, and the address information is usedas the frame number 304 of the feature table 300 which is generated bythe present invention, and a one-to-one synchronization is establishedbetween the video data and the feature table. When the recording ends,the feature table and various variables used in the present inventionare stored in a nonvolatile storage so as to be read and restarted whenthe next recording starts. By doing this, it is possible to newly inputimages, compare them with the images already stored in the video archivein real time at the same time, and automatically associate the samescenes with each other. For example, if a program for comparing theinputted images and the theme song portion is already stored, they aresequential programs and can be automatically collected and arranged as asame classification. If, when sequential programs are watched for thefirst time, information is assigned as a common attribute of the wholesequential programs, it is possible to allow an image just inputted toimmediately share the information. As mentioned previously, also acommercial message appearing repeatedly can be detected and skipped.However, only based on a commercial message existing in an imagerecorded and stored, only a limited number of commercial messages can bedetected. Therefore, even when no images are recorded, images arechecked for 24 hours, and a commercial portion is detected from arepetitive scene, and with respect to the images of the commercialportion, although the images are not recorded, only a feature table isgenerated and recorded. By doing this, more commercial messages can bedetected with the image capacity kept unchanged and a commercial messagecan be skipped more securely. As mentioned above, when the presentinvention is mounted in the next generation video recorder system,automatic arrangement of a recorded program and automatic skipping of acommercial message can be simply executed and the usability is extremelyimproved. In the aforementioned embodiment, it is emphasized thatbroadcasting images can be set as an object. However, needless to say,even images stored in a file may be set as an object.

[0053]FIG. 16 shows an embodiment of a display screen used forinteraction with a user. A film image of video is played back anddisplayed on a monitor window 50 on the display of the computer. As awindow displayed on the same screen, there are a window 52 fordisplaying a list of typical frame images among images, a text window 55for inputting attributes of images and scenes, and a window 54 fordisplaying retrieved results in addition to the window 50. Retrievedresults may be displayed on the window 52. These windows can be moved toan optional position on the screen by operating a cursor 53 which can befreely moved by the mouse which is one of the pointing device 3. Toinput text, the keyboard 4 is used. A typical frame displayed on thewindow 52 is, for example, the top frame of each cut when an image isdivided in cut-wise. Buttons 51 are buttons for controlling the playbackstatus of an image and when the buttons are clicked by the mouse,playback, fast feed, or rewinding of images can be controlled. Scenes tobe played back can be continuously selected by clicking the typicalFrame images displayed as a list on the window 52. In this case, asvideo to be played back, images outputted by the video reproducingapparatus 5 connected to the computer may be used or digitized imagesregistered in an external information storage may be used. When thevideo reproducing apparatus 5 is used, the frame number at the top of ascene is sent to the video reproducing apparatus and the playback isstarted from the scene corresponding to the frame number. When theplayback reaches the frame number at the end of the scene, aninstruction for suspending the playback is sent to the video reproducingapparatus 5. The same may be basically said with a digitized image,though digital video data is read and then it is converted to drawingdata for a computer and displayed as a kind of graphic. When the displayprocess for one frame ends, the display process of the next frame iscontinuously executed and by doing this, moving picture images aredisplayed. In accordance with the time required for the display process,the number of frame images to be displayed for a fixed time is adjustedso as to prevent images from rather fast feed or rather slow feed. Onthe monitor window 50, images from the broadcast receiver 7 can be alsodisplayed.

[0054] The operation procedure for video retrieval by a user using thescreen shown in FIG. 16 will be described hereunder. Firstly, hespecifies an image to be queried. The simplest method is a method forexecuting fast feed or rewinding using the operation buttons 51 andfinding an optional scene by checking images displayed on the monitorwindow 50. The list of typical frames arranged on the window 52 isequivalent to the contents or indexes of a book and by referring to it,he can find a desired scene more quickly. To specify a scene, there isno need to accurately specify the range of the scene and it is desirableto specify an optional frame included in the scene. In this case, it maybe specified by clicking the frame displayed on the monitor window 50 bythe mouse. If a frame image included in the image to be queried isdisplayed in the list of typical frames on the window 52, it may beclicked by the mouse. Next, on the text window 55, the user inputs andregisters attribute information such as the selected scene, title of thewhole image, and person's name from the keyboard. The repetition time ofregistration is optional and if there is no need to reuse the attributeinformation hereafter, there is no need to register the attributeinformation at all. Finally, the user presents a retrieval startrequest. It can be done by clicking the OK button of the text window 55.By doing this, the system starts the retrieval process. The systemimaginarily generates a segment with a fixed length having the specifiedframe just in the middle thereof and applies the segment to theretrieval method of the present invention as an image to be queried. Thetarget image may be newly inputted from the video reproducing apparatus.If it is an image which is already registered as a data base and whosefeature table is generated, the comparison process is performed for thefeature table. In this case, if the frame specified first is included inthe segment of the obtained retrieved result, it is the retrievedresult. Furthermore, it is checked whether it is a partial match or amatch of the whole segment. In the case of a match of the whole segment,it is possible to spread the segment forward and backward and accuratelyobtain the matched segment. This is a retrieving method utilizing theadvantage of the method of the present invention which can search for apartially matched segment at high speed.

[0055] Retrieved results are displayed on the window 54. Displaycontents are attribute information, time information, and others. Or,retrieved results can be graphically displayed in the format shown inFIG. 17. FIG. 17 is an enlarged view of the window 52 and numeral 800indicates an icon image of each typical frame. When a horizontal bar 806is put under an icon image, it is found that a retrieved result existsin the scene corresponding to the icon image. When a retrieved resultspans a plurality of scenes of an icon image, the bar becomes longer forthe part. The bar is classified by a color or a hatching pattern. For aplurality of scenes found by retrieval of the same scene, the same coloris displayed. On the other hand, for a retrieved result of a scene and aretrieved result of another scene, different colors are displayed. Thelist of typical frames can be used as contents or indexes of images asmentioned above and is very useful for finding an image to be queried.However, a dilemma arises that the typical frames are not all imagesincluded in video and if all images are tabulated, it is difficult tofind a desired image from them. Therefore, it can be considered toextract typical characteristics of scenes indicated by the typicalframes by analyzing video and for example, to find video of a part notincluded in images of the typical frames by displaying each icon image800 together with information 802 representing characteristics and timeinformation 804. Such information representing scene characteristicsincludes existence of a person, camera work (zoom, pan, tilt, etc.),existence of special effect (fade in or out, dissolve, wipe, etc.),existence of title, and others. With respect to the image recognitionmethod for detecting images, Japanese Patent Application Laid-Open7-210409 (applied on Aug. 18, 1995) applied by the inventors of thepresent invention can be used. The related disclosure of Japanese PatentApplication No. 7-210409 is incorporated herein by reference. When themethod of the present invention is applied, it can be useful to dissolvethe dilemma of the list of typical frames by another approach. Withrespect to repetitive scenes, not the whole scenes but some of them maybe included in the list of typical frames. For example, in FIG. 18, whenone of the repetitive scenes is clicked and retrieved by the cursor 53,scenes having the same video part as that of the scene are all found andindicated to the user. The retrieved result is indicated in a form ofemphasizing the icon image of the scene including the retrieved segment,for example, like a star mark 810 superimposed on an icon image 808. Inthis case, if the icon image itself to be displayed is replaced with aframe image in the retrieved segment, the indication is made moreclearly understandable. By doing this, if there is only one image of thesame scene as the scene to be found in the list of typical frames, it ispossible to find a desired scene by the help of it and theserviceableness of the list of typical frames is enhanced. The samemethod can be applied to the video displayed on the monitor window 50and it is also possible to specify a frame displayed by clicking,retrieve the same scenes as the scene including the frame, and jump toone of the found scenes. To realize such a process, a troublesomepreparation such as setting of a link node is conventionally necessary.However, if the method of the present invention is used, very quickretrieval is available, so that it is desirable to execute retrievalwhen necessary and no preparation is necessary.

[0056] To execute the self organization process shown in the blockdiagram in FIG. 9, the user does not need to execute any special processfor retrieval and if he just inputs an image, the computer automaticallyexecutes the process.

[0057] In the above explanation, the method for retrieving on the basisof image characteristics of video is described. However, voicecharacteristics may be used and needless to say, to not only video butalso media which can be successively handled, this retrieval method canbe applied.

[0058]FIG. 19 shows an example that the image retrieval art of thepresent invention is applied to a video camera. When power is turned onby a power switch 1961 installed in a process input unit 1960 andpicture recording is instructed by a picture recording button 1962, avoice, image input processor 1910 performs processes of inputting avoice signal from a microphone 1911 and an image signal from a camera1912. The process of the voice, image input processor includes the A-Dconversion process and compression process for inputted voice and imagesignals. A feature extraction unit 1970 extracts frame-wise featuresfrom an inputted image signal. The process contents are the same asthose of the frame feature extractor 106 shown in FIGS. 2 and 9. Theextracted features are stored in a memory 1940 as a feature table. Thememory 1940 uses a built-in semiconductor memory and a removable memorycard. Inputted voice and image signals are retained in the memory 1940,read from the memory 1940 by a playback instruction from a playbackbutton 1963, and subjected to the expanding process for signalcompression and the D-A conversion process by one voice, image outputprocessor, and images are outputted to a display screen 1921, and voiceis outputted from a speaker 1922. A controller 1930 manages and controlsthe whole signal process of the video camera. With respect to aninputted image, the feature thereof is extracted for each frame andstored in the memory. The controller 1930 compares the feature of aninputted image with the features of past frames retained in the memory1940. The comparison process may be performed in the same way as withthe feature comparator 130 shown in FIGS. 2 and 9. As a result ofcomparison, the segment of scenes having a similar feature is retainedin the memory 1940 in the same format as that of the retrieved resulttable (128 shown in FIGS. 2 and 9). Numeral 1950 indicates a terminalfor supplying power for driving the video camera and a battery may bemounted. An image retrieval menu button 1964 instructs a brief editingprocess such as rearrangement or deletion of scenes or a process ofinstructing a desired scene and retrieving and playing back similarscenes by pressing the button 1964 several times on the display screen1921 on which a recorded moving picture image is displayed, for example,like FIGS. 16, 17, and 18. With respect to the art for detecting thechanging point of a moving picture image used for sorting of scenes,Japanese Patent Application Laid-Open 7-32027 (applied on Feb. 21, 1995)applied by the inventors of the present invention can be referred to.The related disclosure of Japanese Patent Application No. 7-32027 isincorporated herein by reference. Scenes are retrieved by use of theimage feature comparison process executed in FIGS. 2 and 9. For such avideo camera, it is necessary to adjust the conditions of the featurecomparison process rather loosely. The reason is that unlike a TVprogram, when a user generally picks up images with a video camera, hescarcely picks up exactly same images. Therefore, when similar scenes orpersons in the same style of dress are photographed in a similar size,the comparison condition is set so that they are retrieved as similarscenes. Picked-up images are analyzed at the same time with recordingand grouping for each scene and indexing between similar scenes arecompleted, so that recorded images can be edited immediately afterpicking up and the usability by a user is improved.

[0059] Effects of the Invention

[0060] According to the present invention, by the aforementioned method,redundant segments with an almost same feature continued are collectedand compared into a unit. Therefore, there is no need to executecomparison for each frame, and the calculation amount can be greatlyreduced, and a form that comparison is falsely executed between thefeature series in frame-wise is taken at the same time, so that themethod is characterized in that the same image segment can be specifiedwith the frame accuracy. Whenever a frame is inputted, only the frame iscompared, so that the processing amount for one frame input is madesmaller and the method is suitable for processing of images requiringthe real time including broadcast images. A plurality of image partsdetected at the same time are exactly same images, so that when they arestored as a set, if a request to search one partial image is presented,the retrieval is completed by indicating another partial image of theset and a very quick response can be expected.

The invention claimed is
 1. A signal series retrieving method in aninformation processing system including time sequential signal inputmeans, a time sequential signal process controller, and a storagecomprising: a step of sequentially inputting time sequential signals; astep of sequentially extracting features in each predetermined period ofsaid inputted time sequential signals; a step of converting saidfeatures sequentially extracted into a feature series corresponding tosaid inputted predetermined period series; a step of compressing saidfeature series in the direction of the time axis; a step of storing saidcompressed feature series in said storage; a step of sequentiallyextracting features from said time sequential signals to be retrieved ineach predetermined period of said inputted time sequential signals; astep of sequentially comparing said features of said time sequentialsignals to be retrieved in each predetermined period with said storedcompressed feature series; a step of storing a progress state of saidcomparison; and a step of retrieving a signal series matching with saidprogress state from said time sequential signals to be retrieved on thebasis of said comparison result between said stored progress state ofsaid comparison and said features of said time sequential signals to beretrieved in each predetermined period.
 2. An image retrieving method inan information processing system including image input means, an imageprocess controller, and a storage comprising: a step of sequentiallyinputting images for each frame; a step of sequentially extractingfeatures from said inputted frame images; a step of converting saidfeatures sequentially extracted into a feature series corresponding tosaid inputted frame image series; a step of compressing said featureseries in the direction of the time axis; a step of storing saidcompressed feature series in said storage; a step of sequentiallyextracting features from said images to be retrieved for each saidinputted frame; a step of sequentially comparing said features of saidimages to be retrieved for each frame with said stored compressedfeature series; a step of storing a progress state of said comparison;and a step of retrieving image scenes matching with said progress statefrom said images to be retrieved on the basis of said comparison resultbetween said stored progress state of said comparison and said featuresof said images to be retrieved for each frame.
 3. An image retrievingmethod in an information processing system including image input means,an image process controller, and a storage comprising: a step ofsequentially inputting images for each frame; a step of sequentiallyextracting features from said inputted frame images; a step ofconverting said features sequentially extracted into a feature seriescorresponding to said inputted frame image series; a step of compressingsaid feature series in the direction of the time axis; a step of storingsaid compressed feature series in said storage; a step of sequentiallyextracting features from said images to be retrieved for each saidinputted frame; a step of sequentially comparing said features of saidimages to be retrieved for each frame with said stored compressedfeature series; a step of storing a progress state of said comparison; astep of updating said stored progress state of said comparison on thebasis of a comparison result with said frame features of said succeedingimages to be retrieved; and a step of retrieving image scenes matchingwith said updated progress state from said images to be retrieved on thebasis of said comparison result between said updated progress state andsaid features of said images to be retrieved for each frame.
 4. An imageretrieving method according to claim 2, wherein as said feature, astatistic of brightness or color is used. wherein as said feature, astatistic of brightness or color is used.
 5. An image retrieving methodaccording to claim 3, wherein as said feature, a statistic of brightnessof color is used.
 6. An image retrieving method according to claim 2,wherein said compression of said feature series in the direction of thetime axis is executed on the assumption that when the difference betweenthe feature of a frame image and the feature of the next frame image iswithin a predetermined tolerance, the features are the same.
 7. An imageretrieving method according to claim 3, wherein said compression of saidfeature series in the direction of the time axis is executed on theassumption that when the difference between the feature of a frame imageand the feature of the next frame image is within a predeterminedtolerance, the features are the same.
 8. An image retrieving methodaccording to claim 2, wherein it is assumed that with respect to a matchwith said progress state of said feature series, when features with morethan a predetermined length match with each other, a comparison resultmatch occurs.
 9. An image retrieving method according to claim 3,wherein it is assumed that with respect to a match with said updatedprogress state of said feature series, when features with more than apredetermined length match with each other, a comparison result matchoccurs.
 10. An image retrieving method according to claim 3, whereinwith respect to storage and update of said progress state of saidcomparison, the number of the top frame in which a comparison match mayoccur is provisionally recorded, and when the comparison matchcontinues, the frame number to be compared is updated, and when thepossibility of comparison match is lost, said frame number to becompared is deleted.
 11. An image retrieving method according to claim2, further comprising: when a match with said progress state of saidcomparison occurs in a plurality of locations, the frame image series insaid plurality of locations is stored so as to be accessed as a relatedset.
 12. An image retrieving method according to claim 3, furthercomprising: when a match with said updated progress state occurs in aplurality of locations, the frame image series in said plurality oflocations is stored so as to be accessed as a related set.
 13. An imageretrieving method according to claim 2, further comprising: when a matchwith said progress state of said comparison occurs in a plurality oflocations, sequential programs on the air are classified on the basis ofthe frame image series in said plurality of locations.
 14. An imageretrieving method according to claim 3, further comprising: when a matchwith said updated progress state occurs in a plurality of locations,sequential programs on the air are classified on the basis of the frameimage series in said plurality of locations.
 15. An image retrievingmethod according to claim 2, further comprising: when a match with saidprogress state of said comparison occurs in a plurality of locations, aspecific image on the air is detected on the basis of the frame imageseries matched in said plurality of locations and the time lengththereof.
 16. An image retrieving method according to claim 3, furthercomprising: when a match with said updated progress state occurs in aplurality of locations, a specific image on the air is detected on thebasis of the frame image series matched in said plurality of locationsand the time length thereof.
 17. A signal series retrieving system in aninformation processor including time sequential signal input means, atime sequential signal process controller, and a storage comprising:means for sequentially inputting time sequential signals; means forsequentially extracting features in each predetermined period of saidinputted time sequential signals; means for converting said featuressequentially extracted into a feature series corresponding to saidinputted predetermined period series; means for compressing said featureseries in the direction of the time axis; means for storing saidcompressed feature series in said storage; means for sequentiallyextracting features from said time sequential signals to be retrieved ineach predetermined period of said inputted time sequential signals;means for sequentially comparing said features of said time sequentialsignals to be retrieved in each predetermined period with said storedcompressed feature series; means for storing a progress state of saidcomparison; and means for retrieving a signal series matching with saidprogress state from said time sequential signals to be retrieved on thebasis of said comparison result between said stored progress state ofsaid comparison and said features of said time sequential signals to beretrieved in each predetermined period.
 18. A signal series retrievingsystem in an information processor including image input means, an imageprocess controller, and a storage comprising: means for sequentiallyinputting images for each frame; means for sequentially extractingfeatures from said inputted frame images; means for converting saidfeatures sequentially extracted into a feature series corresponding tosaid inputted frame image series; means for compressing said featureseries in the direction of the time axis; means for storing saidcompressed feature series in said storage; means for sequentiallyextracting features from said images to be retrieved for each saidinputted frame; means for sequentially comparing said features of saidimages to be retrieved for each frame with said stored compressedfeature series; means for storing a progress state of said comparison;and means for retrieving image scenes matching with said progress statefrom said images to be retrieved on the basis of said comparison resultbetween said stored progress state of said comparison and said featuresof said images to be retrieved for each frame.
 19. A signal seriesretrieving system in an information processor including image inputmeans, an image process controller, and a storage comprising: means forsequentially inputting images for each frame; means for sequentiallyextracting features from said inputted frame images; means forconverting said features sequentially extracted into a feature seriescorresponding to said inputted frame image series; means for compressingsaid feature series in the direction of the time axis; means for storingsaid compressed feature series in said storage; means for sequentiallyextracting features from said images to be retrieved for each saidinputted frame; means for sequentially comparing said features of saidimages to be retrieved for each frame with said stored compressedfeature series; means for storing a progress state of said comparison;means for updating said stored progress state of said comparison on thebasis of a comparison result with said frame features of said succeedingimages to be retrieved; and means for retrieving image scenes matchingwith said updated progress state from said images to be retrieved on thebasis of said comparison result between said updated progress state andsaid features of said images to be retrieved for each frame.
 20. Aprogram storage enabling execution of a process by an informationprocessor including time sequential signal input means, a timesequential signal process controller, and a storage comprising: astorage medium storing a program including the following processes whichcan be read by said information processor; a process of sequentiallyinputting time sequential signals; a process of sequentially extractingfeatures in each predetermined period of said inputted time sequentialsignals; a process of converting said features sequentially extractedinto a feature series corresponding to said inputted predeterminedperiod series; a process of compressing said feature series in thedirection of the time axis; a process of storing said compressed featureseries in said storage; a process of sequentially extracting featuresfrom said time sequential signals to be retrieved in each predeterminedperiod of said inputted time sequential signals; a process ofsequentially comparing said features of said time sequential signals tobe retrieved in each predetermined period with said stored compressedfeature series; a process of storing a progress state of saidcomparison; and a process of retrieving a signal series matching withsaid progress state from said time sequential signals to be retrieved onthe basis of said comparison result between said stored progress stateof said comparison and said features of said time sequential signals tobe retrieved in each predetermined period.
 21. A program storageenabling execution of a process by an information processor includingimage input means, an image process controller, and a storagecomprising: a storage medium storing a program including the followingprocesses which can be read by said information processor; a process ofsequentially inputting images for each frame; a process of sequentiallyextracting features from said inputted frame images; a process ofconverting said features sequentially extracted into a feature seriescorresponding to said inputted frame image series; a process ofcompressing said feature series in the direction of the time axis; aprocess of storing said compressed feature series in said storage; aprocess of sequentially extracting features from said images to beretrieved for each said inputted frame; a process of sequentiallycomparing said features of said images to be retrieved for each framewith said stored compressed feature series; a process of storing aprogress state of said comparison; and a process of retrieving imagescenes matching with said progress state from said images to beretrieved on the basis of said comparison result between said storedprogress state of said comparison and said features of said images to beretrieved for each frame.
 22. A program storage enabling execution of aprocess by an information processor including image input means, animage process controller, and a storage comprising: a storage mediumstoring a program including the following processes which can be read bysaid information processor; a process of sequentially inputting imagesfor each frame; a process of sequentially extracting features from saidinputted frame images; a process of converting said featuressequentially extracted into a feature series corresponding to saidinputted frame image series; a process of compressing said featureseries in the direction of the time axis; a process of storing saidcompressed feature series in said storage; a process of sequentiallyextracting features from said images to be retrieved for each saidinputted frame; a process of sequentially comparing said features ofsaid images to be retrieved for each frame with said stored compressedfeature series; a process of storing a progress state of saidcomparison; a process of updating said stored progress state of saidcomparison on the basis of a comparison result with said frame featuresof said succeeding images to be retrieved; and a process of retrievingimage scenes matching with said updated progress state from said imagesto be retrieved on the basis of said comparison result between saidupdated progress state and said features of said images to be retrievedfor each frame.
 23. An information processor, comprising: a display; amemory having a process program and a data retaining area; control meansfor performing an image input process, an image retrieval process, and adisplay process of a retrieved image on said display according to saidprocess program; means for storing a moving picture image inputted bysaid image input process in said memory in frame wise; means forretrieving a moving picture image segment 2 which is regarded as thesame as a moving picture image segment 1 with a predetermined lengthfrom said memory by the frame series of said stored moving picture imagewhen a frame is newly inputted by said image retrieval process andassociating said moving picture image segments 1 and 2; and means fordisplaying said associated moving picture image segments 1 and 2 indistinction from other moving picture image segments when a movingpicture image inputted by said display process of a retrieved image onsaid display is simply displayed on said display.
 24. A video camera,comprising: a camera for inputting an image; an input processor of saidimage; a storage for storing an image inputted from said camera; aoutput processor for reproducing and outputting an image stored in saidstorage; a feature extractor for extracting the feature of an inputtedimage for each frame; a memory area for tabling and retaining saidextracted feature; a process of comparing the feature of an inputtedimage with the feature on the table; and a process of associating frameshaving the feature agreeing with a predetermined comparison condition assimilar images.